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BOUNDARIES OF VISUAL MOTION 

John M. Rubin k W.A. Richards 

Abstract. A representation of visual motion convenient for recognition 
should make prominent the qualitative differences among simple motions. 
We argue that the first stage in such a motion representation is to make 
explicit boundaries that we define as starts, stops, and force discontinu- 
ities. When one of these boundaries occurs in a motion, human observers 
have the subjective impression that some ileefcing, significant event has oc- 
curred. We go farther and hypothesize that one of these subjective motion 
boundaries is seen if and only if one of our defined boundaries occurs. We 
enumerate all possible motion boundaries and provide evidence that they 
are psychologically real. 
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1 Introduction 

The human visual system is adept at recognizing different kinds of motion. 
We easily identify moving things as walking, waltzing, bouncing, or slith- 
ering, to name just a few types of motion. How do we do it? The visual 
system must somehow represent motions and match these representations 
to descriptions in memory. In this paper, we establish criteria for a motion 
representation and begin its construction. 

We consider only motion of points or blobs, that is, motion without 
shape information. There are three reasons for this approach (see Rubin, 
1985). First, it makes sense to start with the simplest case. As shapes 
are allowed to increase in complexity, the number of degrees of freedom 
of motion skyrockets. Whereas a blob in space can be specified by three 
coordinates, the articulated characters in Lucasfihn's computer-generated 
cartoon, The Adventures of Andre & Wally B., require over five hundred 
parameters of motion. Second, even for an elaborate shape, it seems useful 
to understand as much as possible about its movement as a whole. The 
motion of the center of mass, say, will often be informative, independent 
of the wrigglings of movable parts. The third reason is that the motion 
of parts will eventually be related to some object-centered frame (Wallach, 
1959; Marr k Vaina, 1980). Representing the motion of this frame is an 
important first step. 

2 Representing Motion 

What aspects of motion must a visual system represent if its goal is to 
recognize simple types of motion? A point moving in three dimensions 
is completely described by its position over time, p{t). This representa- 
tion has three shortcomings. First, all that it makes explicit is position, 
and where a motion occurs has nothing to do with what kind of motion it 
is. Second, p{t] is unstable in the sense of Marr k Nishihara (1978): all 
detectable variations in motion are represented independent of their impor- 
tance to motion recognition. By contrast, a stable representation will have 
some explicit component that remains invariant over unimportant changes. 
Finally p{t) depends on the choice of units for measuring space and time. 
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The representation wc develop below overcomes these three objections. 

We seek a motion representation that captures the blatant qualitative 
differences among such motions as bouncing, planetary orbits, and bat 
flight, yet is insensitive to minutia such as the particular value of the viscous 
drag coefficient of air. In pursuit of such a representation, we nmst examine 
in some detail what is meant by a kind of motion. 

Consider the bouncing motion of a tossed ball. What defines bouncing 
is not a particular trajectory, but rather the sequence of free-fall, impact, 
free-fall, impact, and so on (see Forbus, 1981). Intuitively, the trajectory 
is divided into natural parts or eras. Each period of free-fall is an era, as 
indicated in Figure la. Separating two consecutive eras is a brief application 
of force— a bounce — which seems to be a natural motion boundary. Some 
motions, like planetary orbits, lack such boundaries. 

As the foundation of our motion representation, we will define motion 
boundaries in the following section. (Motion eras will be described in Rubin 
(1985); boundaries and eras together will constitute a complete motion 
representation.) Wc will then show that motion boundaries can in principle 
be detected in images from almost any^ viewpoint. 

3 Elementary Motion Boundaries 

Viewers of a bouncing ball perceive fleeting, significant events — subjective 
boundaries— at the bounces (Figure la). Why aren't these boundaries per- 
ceived at the apices, as in Figure lb? Why are subjective boujidaries seen 
at all? An explanation lies in our motivations for the motion boundary 
definitions below. 



3.1 Starts and Stops 

Starts and stops are obvious candidates for motion boundaries, and we de- 
fine them as such. They are illustrated in Figure 2. If they were not made 
explicit in the representation of motion, we would be unable to demarcate 

^ "Ahnost alv/ays" or "almost ;uiy" moans with probability one if the item in question is 
clioscn randomly. 
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Figure 1: A bouncing ball. Circles on the trajectory show the ball's position 
at fixed time intervals, a) Parts of the trajectory as predicted by the theory 
here, and as seen by most observers, b) A possible division of the bouncing 
ball trajectory into parts. 

a period of activity from a period of rest. Starts and stops must be de- 
fined with respect to a reference frame since their definition will require a 
well-defined zero of velocity. There are several clioices for reference frames: 
the viewer, the ground, or moving objects in the scene. Given but a sin- 
gle moving object, the only choice is the viewer's frame. However, for the 
case of articulated shapes, the motion of p^irts will often be most conve- 
niently referred to the motion of the whole, perhaps hierarchically (Marr 
k Nishihara, 1978; Marr k Vaina, 1980). 



3.2 Dynamic Boundaries 

We define a second type of motion boundary that is independent of starts 
and stops: discontinuities of force. A motion boundary based on force, 
unlike starts and stops, will be independent of (inertial) reference frame. 
We choose force discontinuities to supplement starts and stops as motion 
boundaries because discontinuities are robust force events; they can be de- 
tected even in non-inertial frames^ We will henceforth call starts and stops 

^Force discontinuities can be detected in any sniootldy accelerated frame. A non-example 
is a reference frmne tied to a, Jjrownian i)artic]e (Lavenda, 1985). We take "reference 
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Figure 2: Two elementary reference frame boundaries shown in one di- 
mension, and their conjunction, a) Start, b) Stop, c) Pause (see section 
6.1). 

"reference frame boundaries" and force discontinuities "dynamic bound- 
aries." 

There are two advantages to adding force discontinuities as motion 
boundaries. First, the choice captures the intuition that dynamic processes 
are continuous, and that an abrupt change in force is hkely to indicate that 
one process has been succeeded by another. Second, given a rigid body, 
if one point undergoes a dynamic boundary, then almost all ])oints on the 
body must shnultancously undergo dynamic boundaries (see Appendix I). 
This means tliat a visual system can monitor an indiscriminately chosen 
point on a rigid body and still detect dynamic boundaries. 

Steps and Impulses 

We claim that there are two fundiimejiial motion boundaries that are in- 
dependent of the reference framed These are step discontinuities and im- 
pulses of force. Detecting discontinuities involves issues of scale which are 
discussed in Appendix 11. 

For a function / that is continuous at ^o (Thomas, 1972), we have 



/w = ,!ia^(*)-. ';!?-,/(') 



(1) 



frame" to mean an inertial frame, or any frmiic smoothly accelerated with respect to an 
inertial frame. 

"While the arguments in this section {ire given for single- valued functions of time, they 
generalizes straightforwm-dly to vector-Vcilued functions of time. That is, vector-valued 
functions have; the same tAvo elementary types of discontinuity. 
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Figure 3: Two elejiientary types of discontinuity of a single-valued function 
/ of one variable i, and their conjunction, a) A step- change; / can take 
any finite value at ^o. b) An impulse; / at ^o is equivalent to iin impulse 
function, and the limits at ^o are equal. The impulse is depicted by a vertical 
line capped by an arrow to indicate nonfinite value, c) A "stepulse" (see 
section 6.1): / at Iq is equivalent to an impulse, and the limits of / at ^o 
are unequal. 



What ai-e the ways a function can violate (1) and therefore be discon- 
tinuous? Taking / to be force iis a function of time, we will assume the 
right and left limits exist for motion at a biological scale'^ Furthermore, 
we will assume that / is defined at ^05 either taking on some finite value or 
the value of an impulse function^. 

Given the assumptions above, there are only two ways for / to be dis- 
continuous at Iq. One possibility is that the left and right limits of / at ^q 
are unequal. In this case we say / is step-discontinuous at t^. This is the 
first elementary force discontinuity; it is shown in Figure 3a. 

Consider next the case that the left and right limits of / at i^o are equal. 
Then there are two ways for / to be discontinuous at ^o- f{k)) <^a^i have a 
finite value not equal to the value of the limits, or /[to) can have the value 
of an impulse. We claim the former sort of discontinuity is in principle 
undetectable. To see this, note that the choice of a finite value of f{to) 
cannot afTect velocity or position, shice more generally, changing the value 
of a function at isolated points does not affect its integral (Bracewell, 1965). 

^Browiiian motion i3 an example where such limits can fjiil to exist. 

^An imj)iilse function lias value zero except at a single point, yet its integral from 
-00 to 00 is finite find nonz(;ro (Bracewell, 1965). Thus the value of rui impulse is 
not finite. 



It is the remaining subcase in which /{to) has the vahie of an inipnlse that 
is of interest to ns. An impulse is the second elementary force discontinuity; 
it is shown in Figure 3b. 

4 Three-Dimensional Kinematics 

So far we have argued that starts, stops, and force discontinuities should be 
explicit boundaries in the representation of motion. To find these bound- 
ary conditions in the hnage, one must understand their three-dimensional 
kinemeitics. In this section, we will express boundaries as conditions in 
three-dimensional kinematics. Two points must be made with regard to 
this task. First, force, per se, is invisible; we must infer it from the acceler- 
ation it causes. Second, it is inconvenient to seek an impulse discontinuity 
in a (force) function. It is easier to find a step discontinuity in the integral 
of that function (Bracewell, 19G5). Both force discontinuities will therefore 
be expressed in terms of kinematic step discontinuities. 

4.1 Starts and Stops 

Starts and stops will now be precisely defined. Let a reference frame — the 
"scene frame" — be chosen. We have the intuition that an object stops at 
a certain time if it moves for a period prior to that time and is stationary 
for a period after that time. More formally, let s[i) be speed as a fimction 
of time in the scene frame. Define a stop at io when Be > such that 
Vt e (i() - e, ^o), s{t) /- 0, and \/t e [to, k f e), s{t) = 0. The definition of 
a start is analogous. (In practice, measurement of speed will be subject to 
the spatiotemporal resolution hmits of a visual system; see Appendix II.) 

4.2 Dynamic Boundaries 

Consider any kind of force discontinuity. Begin again with Newton's Second 
Law: F(t) = md(t). Note immediately if F[t) is discontinuous^ at some ^O) 



''A vector- valued function of one variable is continuous at a ])oint if its components 
are continuous at that point (Seelcy, 1970). Tlierefore, a vector- valued function has a 
discontinuity at a point if one or more of its components has a discontinuity at that 
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then {md{t)) is also discontinuous at ^o. But mass, at a biological scale, 
does not fluctuate much at all, let alone discontinuously^ Thus given any 
type of dynamic discontinuity, and an assumption of constant mass, a{t) 
must have a discontinuity at ^o. In particular, step discontinuities of force 
bring about step discontinuities in the eicceleration vector. 

We turn next to the case of impulses. Since an impulse of force will 
change the velocity of an object in an instant, an object moving in three 
dimensions that is subject to an nnpulse will undergo a step discontinuity 
of its velocity vector. A simple approach to the result is to note that the 
integral of acceleration is velocity, and the integral of an impulse is a step 
function. Hence, an impulse in force yields a step discontinuity in velocity. 

5 Image Motion 

In this section we examine how the three-dimensional conditions of the 
previous section project to the image. It will be shown below that from 
almost any viewpoint, the three-dim.ensional boundaries will also be two- 
dimensional boundaries in the image. Furthermore, it will be shown that 
a boundary in the image always indicates a three-dimensional boundary; 
that is, there nvc no "false targets". Finally, we show that to find all mo- 
tion boundaries, a visual system must detect exactly four features of image 
motion: ^starts, stops, and step discontinuities of velocity and acceleration. 

5.1 From Three Dimensions to Tv/o 

First we must show that almost any two-dimensional image of a three- 
dimensional boundary contains a boundary. Starts and stops in three- 
dimensions are also starts and stops in the image if the viewer is at rest with 
respect to the scene frame. Likewise, we show in Appendix III that step 

point. 

^We have assumed throughout tliat the mass of the blob if^ coiistmit. If maas is allowed to 
vary, do new motion boundfiries obtain? Continuous vjiriation of mass, as exempHficd 
by a rocket, most of whose mass is fuel, cannot cause an acct^leratiou discontimiity. A 
discontinuity of mass- a break, explosion, or agglomeration will cause ;m accc^leration 
discontinuity. 
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Figure 4: The relations ainong the six types of motion boundaries. 

discontinuities in the three-dimensional velocity and acceleration vectors 
will iilmost always appear as the step discontinuities in the corresponding 
two-dimensional image vectors. 

5.2 False Targets 

Does the appearance of a boundary in the image imply a boundary in three 
dimensions? In Appendix III, we show this is always the case for dynamic 
boundaries. It remains to examine image speed zeroes. In orthographic 
projection, a false target occurs when Vo ~ 5 but v j^ 0. There is thus 
a false target whenever x ^- y ^ /: z. The probability of this exact 
occurrence is zero. Hence false targets will be rare^. (The argument for 
perspective projection is similar.) 



6 Compound Motion Boundaries 

In this section, we consider the possibility of co-occurrence of two or more 
of the quartet of elementary motion boundaries thus far defined. We show 
that the two reference frame boundaries can co-occur, as well as the two 
dynamic boundaries. Furthermore, reference frame and dynamic bound- 
aries are independent. When such combinations are considered, a total of 

^It is also important to fisk wheth(;r the three- dimensional kinematic conditions of Section 
4 could have arisen from circumstances other than the dynfunic boundaries we wish to 
detect. Tlie answer is no b(>cause Newton's Second Law can be considered a definition 
of force. That is, a step discontinuity of velocity is equivalent an impulse of force, and 
a step disconthiuity of an object's acceleration vector is equivalent to a step-change in 
the force on that obj(K:t. 



fifteen mathejuatically distinct motion boundaries emerge. Some of these 
are physically odd. We must emphasize that this set of boundaries does not 
constitute a complete motion representation; eras of motion — periods be- 
tween successive boundaries — must also be described (Rubin, 1985). Also, 
we make no claim that all fifteen compound boundaries are psychologically 
distinct. 

6.1 Conjunctions Within a Boundary Type 

6.1.1 Conjunction of Start and Stop: Pause 

There is an event that is the limiting case of both starts and stops as defined 
in section 4.1. We call this occurrence a pause; it is illustrated in Figure 2c. 
A pause is more formally described as a speed zero such that there are open 
intervals that contain it but no other speed zeroes. Pauses occur naturally 
in simple systems: consider the motion of an inchworm, or a pendulum 
at the ends of its swing. Note that the terms stop, start, and pause are 
mutually exclusive. 

6.1.2 Conjunction of Step and Impulse: "Stepulse" 

A step and an impulse can also co-occur. We call this event a "stepulse;" 
it is illustrated in Figure 3c. A stepulse occurs when a tetherball is struck 
so hard that the tether breaks. Henceforth, the terms step, hnpulse, and 
stepulse will be taken as mutually exclusive. The elementary boundaries 
and their conjunctive progeny are shown in Figure 4. 

6.2 Conjunctions Across Boundary Types 

Starts and stops can coincide with dynamic boundaries. Consider a bean- 
bag striking the ground, an event that involves the coincidence of a stop 
and a stepulse. More generally, at a given moment, velocity (and hence ref- 
erence frame boundaries) and acceleration (and hence dynamic boundaries) 
are independent. To enumerate the combinations, note that at a motion 
boundary, there are four possible dynamic circumstances: step, impulse, 
stepulse, and continuity (no dynamic boundary). Similarly, there are four 
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Table 1: Compound motion boundaries: the fifteen possible motion bound- 
aries, shown as combinations of dynamic and reference frame boundaries 
Exarnplcs of all boundaries are given. TJie common conjunctions are shown 
m Doidface; peculiar combinations are in small font. When there ie neither 
a reference nor a dynamic boundary, tliere is a period of nonzero speed 
with continuously changing force: this is a motion era (Rubin, 1985). 

options for reference frame boundaries: start, stop, pause, and no speed 
zero. Ihere are thus 15 distinct motion boundaries; from 4^ we subtract 
1 for the case m which there is neither a dynamic nor a reference frame 
boundary. These compound motion boundaries are shown in Table 2. All 
15 compound boundaries make mathematical sense, but only some of them 
are physically common. 

To illustrate the notion of compound motion boundaries we examine in 
Figure 5 the four types of start distinguished in l\hh 1. (The following 
argument apphes to stops as well.) Let s{t) characterize speed as a function 
of time, and let s[i) be its derivative. Then the four dynamic possibilities for 
starts are simply described by considering s and s are step-discontinuous at 
the start time ^o- If both functions are continuous, there is no co-occurring 
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Figure 5: Four types of start. Time is plotted on the abscissas; speed s or 
its derivative s are plotted on the ordinates. Start occurs at time U). The 
four possible starts are determined by the continuity of s and s at ^q- 

dynamic boundary. If s is continuous, but not s, then there is a co-occurring 
step discontinuity of force. If s has no step discontinuity, but s does, there 
must be an impulse. Finally, if both s and s are step-discontinuous, a 
stepulse has occurred. 



7 The Trace of Motion 

Above we showed how velocity and acceleration step discontinuities, starts, 
and stops in the image imply motion boundaries. We have yet to explore 
position in the image. That is, can information in the static trace of mo- 
tion in the image be useful in determining what motion boundaries have 
occurred? We show in this section that the answer is yes, given a contin- 
uous trace in the image that has been created continuously^ in time. The 
results are summarized in Table 2. 

7.1 Terminal Points 

First, it is clear that given a generic viewpoint, there is a terminal point in 
the image if and only if there is a terminal point in the three-dimensional 
trace. The moving blob must have had zero speed at the terminal point, 
otherwise the blob would have moved vAt in the next small time interval 

°The correct travers<al of the trace must be specified at a point of self-intersection. 
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/^\j^ 


step discontinuity of force 
or speed zero 



Table 2: Illustration of features of image traces that allow inferences about 
motion boundaries, 'i'he boundary interpretations here are necessary] pos- 
sible boundaries — such as an impulse coinciding with the speed zero at a 
cusp — are not listed. 

At, making the point nonterminaP*\ 

7.2 Cusps and Corners 

First we will define cusps and corners. Next we will show that cusps and 
corners in a three-dimensional curve almost always project orthographically 
to cusps and corners, respectively, in the image. Finally, we will prove that 
cusps and corners necessarily entail motion boundaries. 

^^Anothor possibility is tliat the blob exi)lo(lcd. Wo assume a visufd system would be able 
to detect such a dramatic event mid distinguish it from a speed zero. 
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7.2.1 Definitions 

Let T{s) be the unit tangent vector to the three-dimensional curve, where 
5^ is arclength. At a certain distance Sq along the curve, let f^{si)) and 
T (s()) be the left- and righthand tangent vectors. When the tangent is 
continuous at So, f^ (so) =r f (s,,). By definition, there is a cusp at sq if 
the two tangents are anti-parallel: 'f'(so) = -f (s(,). Finally, there is a 
corner at So if the two tangents are neither parallel nor anti-parallel. A 
concise way of expressing the conditions above is with the dot product 
d = T^ (s()) • T (s()). At a particular place on a curve, if ci = 1, the tangent 
is continuous; i( d = -1, there is a cusp; and if d G (~1,1), there is a 
corner. 

7.2.2 Detection 

In Appendix III we show that, almost always, there is a tangent discontinu- 
ity in the image iff there is a tangent discontinuity in the three-dimensional 
curve. We must show more specifically that image cusps (corners) are 
reliably related to three-dimensional cusps (corners). This is clear: any 
reasonable projection to the image will map a pair of antiparallel vectors 
(at a point in ^^) to a pair of antiparallel image vectors. Furthermore, 
two arbitrary di^ tangent vectors (as in a corner) will almost never map to 
antiparallel image vectors. 

7.2.3 Motion Boundaries 

We interpret cusps and corners in the image by noting that the velocity vec- 
tor of a curve is always parallel to the tangent vector to the trace. Consider 
a corner. We claim that a corner impHes either a speed zero (a reference 
frame boundary) or a step-discontinuity of velocity (force impulse). Sup- 
pose the object has finite speed at the corner. Then speed must change 
instantaneously at the corner, else the trace would have been extended 
along the direction of velocity; that is, there would have been no corner. 
Therefore, there is a speed zero or a force impulse at a corner. 

At a cusp, velocity exactly reverses direction. Thus cusps entail speed 
zeroes. (A cusp could also involve a force impulse, as a corner, but the 
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impulse must be exactly antiparallel to the direction of motion.) 

7.3 Curvature Discontinuities 

We show in Appendix III that (almost always) image velocity and accel- 
eration are continuous iff three-dimensional velocity and acceleration are 
continuous. If p(i) is a plane or space curve, then whenever the curve has 
nonzero speed, curvature is given by (Flanders et ai, 1970, p. 489): 

Curvature of plane and space curves is thus a continuous function of 
velocity and acceleration. By Appendix III, velocity and acceleration in 
the hnage are continuous iff they ai'e continuous in the space curve. But 
then curvature in the image is continuous iff space curvature is continuous. 
Contrapositively, there is (almost always) a curvature discontinuity in the 
image of the trace iff there is a curvature discontinuity in the trace. 

Furthermore, by inspection of (2), we can see that almost all step dis- 
continuities of force (acceleration) bring about discontinuities in curvature. 
Note that force ijupulses cause corners which are infinities of curvature. 
Therefore, almost always, there is a step-cliange of curvature (as opposed 
to a corner, which is an isolated infinity of curvature) iff there is step-change 
of force. 



8 Psychophysical Evidence 

While watching motion, human observers sometimes have subjective im- 
pressions of fleeting, significant events (see again the bouncing ball of Figure 
la). We will call perceived motion boundaries "subjective" and those that 
we have defined mathematically "theoretical." We hypothesize that our 
four elementary theoretical motion boundaries describe a competence of a 
human observer (Chomsky, 1965; see also Yilmaz, 1962 and Marr, 1982); 
the human visual system in principle represents these elementary events. 
(We do not claim the human visual system distinguishes all fifteen com- 
pound motion boundaries.) That is, there is a subjective motion boundary 
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Figure 6: Motions and motion boundaries (marked by heavy slashes) a) 
Planetary orbit — no boundaries, b) Pendulum. The two terminal points 
arc the only boundaries, c) Cydoidal motion: the cusps are motion bound- 
aries, d) The motion of a prolate cycloid: there are no boundaries. 

if and only if there is a theoretical motion boundary. For example, the 
planetary orbit shown in Figure 6a has no theoretical motion boundaries, 
and none are seen. 

We present evidence for our claim below. While it seems to be the 
case that there are subjective boundaries if and only if there are theoretical 
boundaries, it is not clear that the visual system distinguishes the fifteen 
compound boundaries of Table 1. We begin by showing that the human 
visual system is sensitive to aspects of acceleration, a capacity necessary 
for the detection of step-changes of force. 



15 



8.1 Sensitivity to Acceleration 

There is solid evidence to suggest human sensitivity to aspects of motion 
acceleration. A display can be generated of a bouncing ball. A second dis- 
play can be made that creates a trace identical to that of the first display, 
but traverses it at constant speed. The two displays, differing in accelera- 
tion but not average speed, appear strikingly distinct (Rubin, 1985). The 
constant-speed ball seems to whip around the apices of its path. 

Additional evidence that the human visual system is capable of repre- 
senting aspects of acceleration comes from work in a motion extrapolation 
peiradigm. Rosenbaum (1975) showed subjects an object that moved hor- 
izontally at constant acceleration euid disappeared behind a barrier. Sub- 
jects indicated when they thought the object — now no longer in view — 
reached a marked location on the barrier. Results showed that subjects 
extrapolated motion at constant acceleration, as opposed to, say, the av- 
erage velocity of the object while it was visible. In an extension of Rosen- 
baum's work, Jagacinski et al. (1983) found that extrapolated trajectories 
of constant-acceleration were modeled by a period of constant acceleration 
followed by a period of constant velocity. Thus it is clear the human visual 
system is sensitive to some aspects of acceleration; it remains to be shown 
that step-changes and impulse are among these aspects. 

8.2 Reference Frame Boundaries 

Pauses, as in the pendular motion of Figure 6b and the cycloidal motion 
of Figure 6c, ai'e subjective boundaries. If a constant horizontal velocity is 
added to a cycloid, prolate and ctirtate cycloids obtain. These new trajec- 
tories have no pauses, and — when the trajectories are sufficiently distinct 
from a cycloid — no motion boundaries are seen (Rubin, 1985). See Figure 
6d. 

The perception of starts has been examined by Runeson (1974, 1977). 
He presented a Vciriety of starts differing in how speed changed as a func- 
tion of time^^ Runeson argued that human perception distinguishes two 
sorts of starts. Undramatic starts are seen when speed increases smoothly 



^^ All of Rimeson's displays were of linear motion. 
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from zero to some asymptotic velocity. Otherwise, dramcatic or eventful 
starts are seen. Perception of undramatic starts involves simply a sensa- 
tion of velocity and its inception. Eventful starts are perceptually more 
complex: observers report something happening at the beginning of motion 
distinct from their sensation of velocity. Runeson's results are consistent 
with our claim that starts cause subjective motion boundaries. However, 
our scheme distinguishes four types of starts to Runeson's two. Whereas 
all of our start types arc motion boundaries, they are not all perceptually 
distinct: stepulse and impulse starts are dramatic; smooth and step starts 
are uneventful. (The perception of stops is analogous; stepulse and impulse 
stops are violent, whereas smooth and step stops are peaceful.) 

8.3 Dynamic Boundaries 

Force impulses, as exemplified by bounces, are seen as motion boundaries. 
To examine the appearance a step-change of force, we created a display of 
a ball moving with constant acceleration in one direction. When the di- 
rection of acceleration was suddenly switched, the change was conspicuous 
to observers as a motion event. Furthermore, force step-changes are per- 
ceptually distinct from a force impulses; the former are legato, the latter, 
stacatto. 



9 Discussion 

9.1 Properties of Motion Boundaries 

To be useful for recognizing different kinds of motion, a motion represen- 
tation should be psychologically relevant, mathematically convenient, and 
physically apropos. Psychological relevance means that the primitives of 
the representation should be computable by the human visual system, and 
the scheme should divide the class of trajectories into roughly the same 
equivalence classes as human observers. Our treatment of motion bound- 
aries satisfies this criterion. Motion boundaries aje the transient aspect of 
a complete motion representation, the ongoing aspect of which is eras (Ru- 
bin, 1985), descriptions of the periods between successive boundaries. The 
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human visual system is sensitive to a few motion events, and preliminarily, 
it seems that our theoretical boundaries axe at worst a refinement of the 
psychological classification of motion transients'^. 

A mathematically convenient representation is one that has useful in- 
variant properties. Our motion boundaries, based on local properties of 
p(f), the description of three-dimensional position as a function of time, 
exhibit three useful invariances. The motion boundaries do not depend 
on the units for measuring space; they are invariant over spatial scaling. 
Also, if a given motion is repeated twice as fast, the boundaries main- 
tain their relative positions: this is speed scaling. Finally, the motion 
boundaries are transparent. A local feature of a three-dimensional curve is 
said to be "transparent" when there is an associated feature in the image 
of the curve — call it the "shadow" of the three-dimensional feature — such 
that whenever the shadow is found in the image, the feature is guaranteed 
present in the world, and whenever the feature is present in the world, 
images of the curve from almost all viewpoints contain the shadow. (See 
Appendix IV.) Stated more simply, a transparent feature is one that can 
be found without error from almost any viewpoint. 

The result is even stronger than just stated: not only do the motion 
boundaries have useful invariant properties; they are the only reasonable 
local properties of motions having those invariances. Consider the class of 
local properties of curves that are invariant over spatial scaling. This class 
consists of zeroes and impulses of a curve and its derivatives. This is the 
same class as that of local curve properties invariant over speed (or force) 
scaling. It is also the same as the class of transparent properties. Each of 
these three types of invariance thus independently specifies the same class of 
local properties. (See Appendix IV.) Furthermore, our motion boundaries 
are the lowest-order members of this class'*^. 



^^Let X and 1/ be seta of sets that partition a universe U into equivalence chisses. Then 
X is said to be a refinement of 1/ if the inend)ers of a member of X are members of 
exactly one member of I/. A refinement of a chissificatory system tlius makes all the 
original distinctions, and then some. Cr\uial to the notion is the fact tliat refinements 
do not carve up the universe in an indtipendent way. 

^^An exami)le of a higher-order term that is not a motion bound;iry is an impulse in the 
eleventh dc^rivative of the curve. 
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There is no a priori reason that mathematically convenient boundaries 
should be physically meimingful. In fact, the higher- order members of 
the class of invariants described in Appendix IV (impulses in the eleventh 
derivative of position, say) are probably without physical importance. But, 
as we have argued, the lower-order members— starts, stops, and dynamic 
discontinuities — signify meaningful force events. 

9.2 Relation to Previous Work 

Some of our suggestions for the representation of motion have been made 
by others. Our work is distinguished from previous motion studies by the 
following combination of features. First, the scope of our representation is 
large, encompassing any piecewise-continuous motion of any shape that can 
be construed as a point or blob. Second, the definition of motion bound- 
aries is founded in the physics of the macroscopic world; our representation 
makes force explicit. Third, we show rigorously that our theoretical motion 
boundaries iire in one-to-one correspondence with certain kinematic image 
conditions, and, more importantly, are the lowest order members of the 
class of reliably detectable local properties of motion. Finally, our scheme 
is "bottom-up;" no knowledge of the shape or motion of particular objects 
is necessary. 

Gibson (1979, p. 101) advanced the idea that motion can be divided 
into natural parts, writing ". . . the flow of ecological events consists of 
natural units" that arc arranged hierarchically so that "[w]hat we take to 
be a unitary episode is therefore a matter of choice . . . ." In contrast, we 
suggest that motion boundaries are rigidly defined. (We do not rule out 
the possibility of description at two or more scales; see Appendix II.) 

Runeson (1977) focused on how material properties of objects (relative 
mass, elasticity, and so on) could be inferred from kinematics. While Rune- 
son would be interested, say, in inferring the elasticity of a bouncing ball, 
we are primarily interested in recognizing bouncing motion. Though his 
goal differs from ours of motion recognition, his distinction between events 
and processes is similar to our division between continuous motion eras and 
transient motion boundaries. Events for Runeson are abrupt, evanescent 
occurrences that signify energy transfer; processes are enduring kinematic 
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goings-on. Runeson's events seem related to the dynamic snbset of motion 
boundaries in this paper. ^'^ Runeson argued that a perceptual system must 
give priority to dynamic events over processes because the former are more 
informative about causal relations. By contrast, we give equal weight to 
dynamic and reference frame boundaries in our motion representation. 

Forbus (1981) undertook to describe complex motions by what he called 
an "Action Sequence." An Action Sequence is a concatenation of Acts, 
each of which is a period (or a moment) of a single type of motion that can 
be described by a particular equation. Bouncing motion is, for example, 
represented as FLY UP, FLY DOWN, COLLISION, FLY UP, FLY DOWN, 
and so on. 

Some important differences between Forbus's work and ours must be 
noted. Forbus was interested in reasoning about motion from diagrams or 
word problems; we are interested in perception. Forbus used a restricted 
two-dimensional domain in which gravity is the only force. Furthermore, his 
program only reasoned about Acts that are on a menu, namely, Collide and 
Fly. The scope of our theory is greater; any sort of (piecewise-continuous) 
force in three dimensions is acceptable, and more importantly, our scheme 
does not rely on a menu of force equations; novel forces can be represented. 

Two motion representations have been proposed to describe the mo- 
tion of complex, articulated shapes. Laban (1975) developed an elaborate 
notation for transcribing choreography. "Labanotation," as it is called, is 
necessarily specific to the human form. Marr k Vaina (1980) offered a 
means of representing the motion of objects that admit 3D model descrip- 
tions. These two schemes thus have narrower scope tlian ours since they 
apply only to certain classes of shapes. Moreover, their descriptions relate 
the motion of parts to superordinate parts or the whole, paying less atten- 
tion to the overall motion of the whole. (Marr k Vaina suggested without 
motivation that the representation of the motion of the entire 3D model 
mark speed zeroes and step discontinuities of velocity.) 

Badler (1975) took a top-down approach to the description of motion 
based on a sequence of static drawings. That is, his scheme required that 
objects be recognized so that knowledge about them can be used in the mo- 

^^ Events arc not precisely defined in Rxineson's work. He seems to liave velocity but not 
acceleration step discontinuities in mind. 
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tion description. By contrast, we suggest a visual system extract the most 
informative possible description of motion independent of object recogni- 
tion. Badler did, however, suggest that the representation of motion make 
explicit an assortment of conditions that included starts, stops, and trajec- 
tory discontinuities. 

10 Summary 

We have proposed that blob motion be cut into eras at certain natural 
boundaries. The eras can then be given qualitative descriptions (Rubin, 
1985). This approach to representation for recognition is reminiscent of 
Hoffman k, Richards' (1982,1984) work on static planar contours. They cut 
contours at a boundary condition and then describe the resulting natural 
parts qualitatively. A significant difference, however, is that the motion 
boundaries of this paper are in themselves meaningful; Hoffman k Richards' 
contour boundary condition serves only to separate parts. 

We began by noting criteria for a motion representation suitable for 
recognition: stability over niggling variations in trajectory, and invariance 
over space and time scaling. We defined two types of motion boundaries 
that satisfy the two criteria above. Dynamic boundaries (force discontinu- 
ities) are a good foundation for a motion representation in that their ap- 
pearance in the image is reliably related to events in the three-dimensional 
world, regardless of the (smooth) motion of the observer. Reference frame 
boundaries — starts, stops, and pauses — depend on the viewer's frame. W^e 
gave evidence that our theoretical motion boundaries underlie subjective 
boundaries. 

Appendix I: Rigid Bodies and Impulses 

Claim. If a rigid body is subjected to a force impulse, then almost all its 
points will move as if they have been subjected to a force discontinuity. 

Proof sketch. Any three-dimensional motion of a rigid body is equiv- 
alent at each instant to a unique twist (Coxeter, 1961). A twist is defined 
by cin axis /, an angular velocity w about l, and a translational velocity v 



21 



along I. An impulse will cause a discontinuity in at least one of /, w, and v. 
Case I: The impulse changes translation v instantaneously (and possibly 
uj as well). Then all points are affected, since translations have no invariant 
points. Case II: The impulse affects the rotation of the object (but not 
its translation). All points on the rigid body that He on / are invariant, 
and will not show the impulse. But for a two- or three-dimensional rigid 
body, such invariant points constitute a measure-zero set. Hence there is 
zero probability of choosing an invariant point on the body at random. 
Case III: The impulse changes the twist axis I instantaneously into a new 
axis I' . Then the only invariant points on the rigid body will be those that 
lie on the intersection of / and I'. By Case II, there is zero probability of 
selecting such a point. 

Appendix II: Scale Problems 

Here we describe (but do not solve) some problems of scale that affect the 
detection of motion boundaries. These problems are analogous to scale 
issues that arise when trying to represent static figures. 

Resolution 

Any visual system that detects starts will have to face the following problem 
of scale. Let speed be given by s{t) = \ sin || for, say t G (0, l]. Note there 
are an infinite number of values of t e (0, 1] that satisfy the definition of 
start. The number of starts actually perceived will be finite and depend on 
the spatiotemporal resolution of the system. Analogously, if one inspects 
(the static) graph of the function s{t), one will see only a finite number of 
points where the graph of the function touches the x-axis, depending on 
the spatial resolution of the human visual system. 

Discontinuity 

The detection of dynamic discontinuities is a scale problem that is not 
solved simply by knowing the spatiotemporal resolution of a visual system. 
Consider the analogous static problem of deciding when a continuous curve 
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has a corner. That is, when does the tangent change rapidly enough (with 
arclength) to be considered discontinuous? Clearly no fixed "A tangent" 
will suffice; rather, the critical value seems to depend on how rapidly the 
tangent is changing in a neighborhood. We expect similar considerations 
to apply to the detection of velocity and acceleration step discontinuities. 

Description of Independent Scales 

Consider how the human visual system might represent the shape of a tire. 
It is reasonable to suspect the description has at least two distinct scales 
(see Mandelbrot, 1977). At the larger scale, the overall rounded torus is 
described; at the smaller scale, the terrain of the tread is represented. Cer- 
tain motions will also be best described at two separate scales. Consider 
the motion of a reaching hand: it moves through space along a smooth arc. 
Looking closer, one might notice the hand is trembling. These two sorts 
of motion are independent— independent in spatial and temporal scale and 
independent in cause. Note that the representation of the tremble might 
have motion boundaries (the pauses where the oscillation reverses direc- 
tion), whereas there might be no boundaries in the larger scale description. 
The punchline is that, for complex motions, motion boundaries must be 
sought at a particular scale, and that descriptions at two or more scales 
might be necessary. 

Appendix III: Images of Discontinuities 

Below we show that images of continuous curves are continuous. It is 
intended that p be interpreted either as the position (section 5), the tangent 
(section 7.2), or the acceleration vector (section 7.3) of a three-dimensional 
curve as a function of time, and / is any projection fimction that maps 
space to aji image plane such that the pre-image of every miage point is 
a one-manifold in space (a generalized "line of sight"). It is clear that 
orthographic and perspective projection are reasonable in this sense. 

Claim. Let p(t) : 3?^ i-> 9?^ and / : 3?^ h-> di^ be continuous func- 
tions such that I'^(x,y) is a one-manifold in 9?^ (i.e., rank(Jacobian(I))— 2). 
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Then p{t) is continuous => I{p{t)) is continuous, and I(p{t)) is continuous 
=> (almost always) that p{t) is continuous. 

Proof. Since the composition of continuous functions is continuous 
(Seeley, 1970), we have immediately that the continuity of p implies the 
continuity of I(p). Next, suppose that at some to, I{p{to)) is continu- 
ous, but that contrary to the claim we are to prove, p is discontinuous at 
U). We consider only step discontinuities. Let p* (<o) = Hm^ yto\P{t), and 
p [to] = lmit->t,, p{t). A step discontinuity at ^o implies p'(i()) ^p [Iq). 
By continuity of /(p), we know /(p' (^o)) = I{p {k)). But / assigns the 
same Sff^ y^lue only to points in 9?^ lying on a particular one-manifold. 
There is zero probability that the two points p* (i^o) and p (io) lie on one 
of those special one-manifolds. 

Corollary. Let p : dt^ ^ di^ he a vector-valued function, and let / 
be a reasonable and continuous imaging function as before. Then, almost 
always, /(p) is discontinuous at to iff p is discontinuous at to. 

Proof. (Contrapositive of claim above.) 

Appendix IV: A Unique Class of Boundaries 

In this appendix we investigate reJiably detectable local properties of space 
curves parameterized by time. Reliably detectable events are those that 
can be found with a 100% hit rate and no false targets. More specifically, 
a reliably detectable curve event is one that is associated with a particular 
image feature, such that almost all images of the curve possess that feature, 
and whenever that feature appears in an image, it is certain the space curve 
event has occurred. We are particularly interested in reliably detectable 
events that are invariant over speed and space scaling. 

We will show that and impulses of the derivatives of a space curve are 
the only reliably detectable events. The definitions and chiims that follow 
serve more to make precise the special properties of and impulses than 
to derive surprising mathematical results. 

We begin by describing a moving point. 

Definition. A position curve in 9?" {PCr^) is a continuous function 
p : 3ff f-^ 9?", n > 2. 
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We would like to generalize the notion of function in order that all 
position curves be differentiable infinitely many times (see Lighthill [1958] 
for a more elaborate generalization of functions). To this end we add a 
value "*" to the range of real vector-valued functions, where * represents 
infinity, as in an impulse. The following definition assures that the values 
* are sparse. 

Definition. A piecewise real curve [PRCn) is a function h : 9? i— > 
3?" U {*}, n > 2, such that Vr* G 9? such that h{r*) =:::*, Be > such that 
Vr G (r* - e,r*) U {r\r* |- e), h{r) ^ *. 

It is clear* that a PC is a FRC. We next define the "generalized deriva- 
tive" of a PRC so that impulses are treated correctly (see Bracewell, 1965). 
That is, the generaHzed derivative at a step or stepulse is an impulse; the 
generahzed derivative at an impulse is 0; and the derivative at an ordinary 
point matches the normal definition of derivative. 

Definition. h^^\ the i*^ generalized derivative of a PRCn h is given 
recursively by 

[ * if lim, ..f ( iS^- i)(r) ^ lim, .,^ '\r) 
Vfe^R, h^\r) =. \ if/i(^'^)(f)=:* 

[li,„^^.^^^M.^ilirul otherwise 

where h^^^' denotes h. The following is an immediate consequence of the 
definition above: 

Claim. The generalized derivative of a PRC is a PRC\ hence a PRC 
is infinitely generafized-differentiable. 

We now discuss orthographic projections / : 9?^ h-> 3?^. We must state 
precisely how the projection is related to the coordinate frame of the po- 
sition curve. We want to rule out the troublesome possibility of / varying 
erriitically with r, as would be the case of a reference frame tied to a jack- 
hammer. Any / can be exactly specified by six numbers: three rotations a, 
/?, and 7, and the location of the image origin (a, 6, c). We will call / an aris- 
totelian view or projection of a PC'i when 0— ^==^ = $^ = 4^ = 4-==^. 

'^ '' «* dr dr dr dr dr dr 

Simply put, an tiristotelian view is stationary with respect to the world ref- 
erence frame. 

Next we prove a claim about reasonable images of position curves. 
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Claim. Given an aristotelian projection / : 9?"^ f-> 31?^ and a PC3, p, 
then I{p) : 9? h-^ 9f?Ms a PC2. Hence, I^^\p), the f'^ generalized derivative 
of /(p), exists for all j. 

Proof. First note that I{p) is the composition of continuous real-vahied 
functions emd is therefore continuous cind real-valued itself. It is, therefore, 
a PC2. But PCs are infinitely generahzed-differentiable, so the claim is 
proved. 

We now want to formalize the idea of a reliably detectable event. Below, 
the local event of the curve to be detected is some particular value po (the 
only strictly local property of a function is its vidue); its associated image 
feature is iq. 

Definition. A j-transparent value of a PCz-, p, under aristotelian view 
is any po G 9?^ U {*} such that li^ G 9?^ U {*} such that for all aristotelian 
projections / : U^ k> 9?^ and for all r, p^^'>[r) - po <=^ I^^'^{p{r)) - zq. 

Definition. A transparent value of a PC^ under aristotelian view is a 
j-transparent value from some j. 

We are now ready for the first major claim. 

Claim. The only transparent values of a three-dimensional position 
curve under aristotelian view are {*,0}. 

Proof. Consider the second generalized derivative of the position curve. 
We claim * is transparent; this follows from Appendix III, where we show 
that images of discontinuities are almost always discontinuities, and from 
the fact that * is the derivative of a step-discontinuity. We claim that 
is transparent due to the restriction to aristotelian view. The second 
derivative of the change of viewpoint is zero, hence on the curve maps 
to in the image, and almost always, 6 in the image arises from in the 
space curve. It remains to show that other non-zero, non-impulse values 
of curves cannot be transparent. This follows because the orientation of a ' 
aristotelian image with respect to three-space is arbitrary (though fixed). 
The only vector invariant over rotation is 0, but transparency quantifies 
over all ai-istotelian views. 

We next investigate another sort of invariance. 

Definition. A scaling of a PRC^ h is a function 



^^^ ~ I * if h(r) = * 
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where a. is some constant. 

For a given position curve, suppose all velocities are doubled. The 
resulting motion is a scaling of the original motion. So is the motion that 
obtains when all forces (accelerations) are multiplied by some constant. 

Definition. A scalable value of a PRC^ h is any po G 9?" U {*} such 
that Vr E 9fJ and for all scalings g of h, h{r) — po <==>- gif) — Po 

We state the following without proof. 

Claim. A PRCn can have at most two scalable values, {*,0}. 

We have shown that transparency (under aristotelian view) and scala- 
bility independently select special values * and 0. Note that a scaling of 
the position curve is equivalent to a change of spatial units. Scalings of the 
first and higher generalized derivatives of the position curve aic speed and 
force scalings, and so on. 
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